1. Field of the Invention
The subject disclosure relates to methods and systems for archiving, mirroring/replicating, or backing up information in a limited bandwidth distributed computing network, and more particularly to replicating/mirroring data while minimizing communication traffic and without impacting application performance in a redundant array of independent disks (RAID) array.
2. Background of the Related Art
Remote data replication or archiving data has become increasingly important as organizations and businesses depend more and more on digital information. Loss of data at the primary storage site, for any reason, has become an unacceptable business risk in the information age. Since the tragic events of Sep. 11, 2001, replicating data to a remote storage back-up site has taken on new urgency as a result of heightened awareness of business resiliency requirements. Remote data replication is widely deployed in industry as varied as finance, legal and other corporate settings for tolerating primary failures and disaster recovery. Consequently, many products have been developed to provide remote replication or mirroring of data.
One type of remote replication product is block-level remote mirroring for data storage in fiber channel storage area networks (FC-SAN). Block-level remote mirroring is typically done through dedicated or leased network connections (e.g., WAN connection) and managed on a storage area network based on FC-SAN. EMC Corporaton of Hopkinton, Mass. offers such a product know as the Symmetrix Remote Data Facility
In particular, use of RAID disk drives has also been widely used to reliably store data for recovery upon failure of the primary storage system. However, replicating data to a geographically remote site demands high network bandwidth on a wide area network (WAN). It is well-known that high bandwidth WAN connections such as leased lines of tens or hundreds of megabytes are very costly. As such, use of such communication networks is limited to companies that can afford the expense. In order to enable remote data replication over commodity Internet connections, a number of technologies have emerged in the storage market. These technologies can be generally classified into three categories: WAN acceleration using data compressions; backup changed data blocks (delta-blocks); and backup changed bytes using byte-patching techniques.
Compression attempts to maximize data density resulting in smaller amounts of data to be transferred over networks. There are many successful compression algorithms including both lossless and lossy compressions. Compression ratio ranges from 2 to 20 depending on the patterns of data to be compressed. While compression can reduce network traffic to a large extent, the actual compression ratio depends greatly on the specific application and the specific file types. Although relative lightweight real-time compression algorithms have had great success in recent years, there are factors working against compression algorithms as a universal panacea for data storage. These factors include high computational cost, high latency, application or file system dependency, and limited compression ratio for lossless data compression. There are also technologies that replicate or mirror changed data in a file reducing network traffic. These technologies work at a file system level. The draw back of technologies working at the file server level is that they are server intrusive because installation is required in the file system of the server. As a result, the limited resources of the server (such as CPU, RAM, and buses that are needed to run applications) are consumed. In addition, such file system level technologies are file system dependent.
Mirroring changed data blocks (i.e. delta-blocks) reduces the network traffic because only changed blocks are replicated over the network. Patching techniques find the changed data between the old version and the new version of a file by performing a bit-wise exclusive OR operation. While these approaches can reduce network traffic, significant overhead is incurred while collecting the changes. To back up changed data blocks, the system has to keep track of meta-data and to collect changed blocks from disks upon replication. To back up changed bytes of a file, a process of generating a patch and comparing the new file with the old file, has to be initiated upon replication. The generation and comparison process takes a significant amount of time due to slow disk operations. Therefore, these technologies are generally used for periodical backups rather than real-time remote mirroring. The recovery time objective (RTO) and recovery point objective (RPO) are highly dependent on the backup intervals. If the interval is too large, the RPO becomes large increasing the chance of losing business data. If the interval is too small, delta collection overheads increase drastically slowing down application performance significantly.
The lower cost solutions also tend to have limited bandwidth and less demanding replication requirements. For example, the lower cost solutions are based on file system level data replication at predetermined time intervals such as daily. During replication, a specialized backup application program is invoked to collect file changes and transfer the changes to a remote site. Typically, the changes may be identified by review of file meta data to identify modified files. The modified files are then transmitted to the server program through TCP/IP socket so that the server program can update the changes in the backup file. It can be seen that such approaches are more efficient than backing up every file. However, data is vulnerable between scheduled backups and the backups themselves take an undesirably long amount of time to complete.
Several following examples, each of which is incorporated herein by reference in its entirety, disclose various approaches to parity computation in a disk array. U.S. Pat. No. 5,341,381 has a parity cache to cache RRR-parity (remaining redundancy row parity) to reduce disk operations for parity computation in a RAID. U.S. Pat. No. 6,523,087 caches parity and checks for each write operation to determine if the new write is within the same stripe to make use of the cached parity. U.S. Pat. No. 6,298,415 caches sectors and calculates parity of the sectors in a strip in cache and reads from disks only those sectors not in cache thereby reducing disk operations. These prior art technologies try to minimize computation cost in a RAID system but do not solve the problem of communication cost for data replication across computer networks. U.S. Pat. No. 6,480,970 presents a method for speeding up the process of verifying and checking of data consistency between two mirrored storages located geographically remote places by transferring only a meta data structure and time stamp as opposed to data block itself. Although this prior art method aims at verifying and checking data consistency between mirrored storages, it does not consider solving efficiently transferring data over a network with limited bandwidth for data replication and remote mirroring.
In view of the above, a need exists for a method and system that mirrors, replicates and archives data in real-time while minimizing the burden on the communication lines between the primary storage site and the mirror storage facility.